[ResponseOps] Cases analytics index synchronization by adcoelho · Pull Request #222820 · elastic/kibana

adcoelho · 2025-06-05T13:40:16Z

Closes #221232

Merging into a feature branch

Summary

This PR adds:

Synchronization task registration.
Synchronization task scheduling.
Testing

How to test

A few things are needed.

Create a role that has index creation permission. Below is what works for me; it might be overkill.

Create a user with the role created in step 1. Let's call the user index_creator. The password should be changeme.

Start Kibana with this user as the Elasticsearch user - yarn start with the option --elasticsearch.username=index_creator.
You can check the content of these indexes in the Dev Tools.

GET /.internal.cases/_search
GET /.internal.cases-comments/_search
GET /.internal.cases-attachments/_search

Create Cases, Comments, and Attachments.
After about 5 minutes, check again the content of the indexes using the instructions in step 7.
Confirm the Cases, Comments, and Attachments created in step 8 can be seen in the analytics indexes.

elasticmachine · 2025-06-10T12:28:22Z

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine · 2025-06-10T12:28:23Z

Pinging @elastic/response-ops-cases (Feature:Cases)

adcoelho · 2025-06-12T12:03:07Z

The failing functional tests with [.internal.cases-attachments] Failed to create the index. Error message: security_exception are happening because the internal user does not yet have permissions for those indexes.

ersin-erdal

LGTM
Just added a question about number of reindexing attempts.
And functional test would be good for this. We can just create a case in the test and expect it to be reindexed in the analytics index.

cnasikas

Great work!! LGTM! I left some comments. Most of them are nits.

cnasikas · 2025-06-13T11:03:17Z

+    });
+  });
+
+  describe('Error handling', () => {


For tests with try/catch, we should put it at the beginning of the test expect.assertions(1), so the test will fail if the assertion in the catch is not called for some reason.

cnasikas · 2025-06-13T11:04:53Z

+        expect(isRetryableError(e)).toBe(null);
+      }
+
+      expect(logger.error).toBeCalledWith(


Nit: I think expecting certain error messages can lead to tests that are not resilient to refactoring. For example, every time we make a small change in the message, we should also update all tests. What about just checking expect(logger.error).toHaveBeenCalled();

elasticmachine · 2025-06-13T14:18:04Z

💔 Build Failed

Buildkite Build
Commit: 08dc5cc

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #71 / alerting api integration security and spaces enabled - Group 2 Alerts alert deletion schedule alert deletion space_1_all at space1 should delete the correct of alerts - all category inactive alerts
[job] [logs] Jest Tests #14 / SynchronizationTaskRunner Error handling An error is thrown for invalid task state
[job] [logs] Jest Tests #14 / SynchronizationTaskRunner Error handling An error is thrown for invalid task state
[job] [logs] FTR Configs #118 / task_manager check_registered_task_types should check changes on all registered task types
[job] [logs] FTR Configs #118 / task_manager check_registered_task_types should check changes on all registered task types

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [fa8ef65]

History

cc @adcoelho

This PR is for a feature branch that is being merged into main. The relevant PRs are: - #219211 - #222820 - #223241 - #224388 - #224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co> (cherry picked from commit e566fec) # Conflicts: # x-pack/platform/plugins/shared/task_manager/server/mocks.ts

This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

adcoelho self-assigned this Jun 5, 2025

adcoelho added Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// Feature:Cases Cases feature labels Jun 5, 2025

adcoelho requested a review from ersin-erdal June 5, 2025 13:48

adcoelho marked this pull request as ready for review June 10, 2025 12:28

adcoelho requested a review from a team as a code owner June 10, 2025 12:28

adcoelho added 4 commits June 11, 2025 11:07

Initial commit.

d6cf852

Updated synchronization logic to handle errors.

3ad700e

Sync task tests.

7b1d129

wait_for_completion

050f701

adcoelho force-pushed the cai-synchronization branch from 707b861 to 050f701 Compare June 11, 2025 09:08

Fix linting.

5133a0b

adcoelho force-pushed the cai-synchronization branch from 9946246 to c393168 Compare June 12, 2025 07:27

ersin-erdal approved these changes Jun 12, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/cases/server/cases_analytics/tasks/synchronization_task/index.ts

Comment thread ...hared/cases/server/cases_analytics/tasks/synchronization_task/synchronization_task_runner.ts Outdated

cnasikas approved these changes Jun 13, 2025

View reviewed changes

Fixed tests and addressed PR comments.

d97eeeb

adcoelho force-pushed the cai-synchronization branch from c393168 to d97eeeb Compare June 13, 2025 12:22

Addressing final PR comments.

08dc5cc

adcoelho merged commit 7b1e564 into elastic:cases-analytics-index Jun 13, 2025
8 of 10 checks passed

adcoelho mentioned this pull request Jun 17, 2025

[ResponseOps] Cases analytics index #223405

Merged

Conversation

adcoelho commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Uh oh!

elasticmachine commented Jun 10, 2025

Uh oh!

elasticmachine commented Jun 10, 2025

Uh oh!

adcoelho commented Jun 12, 2025

Uh oh!

ersin-erdal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cnasikas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnasikas Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

cnasikas Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adcoelho commented Jun 5, 2025 •

edited

Loading

ersin-erdal left a comment •

edited

Loading

elasticmachine commented Jun 13, 2025 •

edited

Loading